In the rapidly advancing field of artificial intelligence, machine perception is becoming paramount to achieving increased performance. Image classification systems are becoming increasingly integral to various applications, ranging from medical diagnostics to image generation; however, these systems often exhibit harmful biases that can lead to unfair and discriminatory outcomes. Machine Learning (ML) systems that depend on a single data modality--i.e., only images or only text--can exaggerate hidden biases present in the training data, if the data is not carefully balanced and filtered. Even so, these models can still harm underrepresented populations when used in improper contexts, such as when government agencies reinforce racial bias using predictive policing. This thesis explores the intersection of technology and ethics in the development of fair image classification models. Specifically I focus on improving fairness and methods of using multiple modalities to combat harmful demographic bias. Integrating multimodal approaches, which combine visual data with additional modalities such as text and metadata, allows this work to enhance the fairness and accuracy of image classification systems. The study critically examines existing biases in image datasets and classification algorithms, proposes innovative methods for mitigating these biases, and evaluates the ethical implications of deploying such systems in real-world scenarios. Through comprehensive experimentation and analysis, the thesis demonstrates how multimodal techniques can contribute to more equitable and ethical AI solutions, ultimately advocating for responsible AI practices that prioritize fairness.

Contrastively trained text-image models have the remarkable ability to perform zero-shot classification, that is, classifying previously unseen images into categories that the model has never been explicitly trained to identify. However, these zero-shot classifiers need prompt engineering to achieve high accuracy. Prompt engineering typically requires hand-crafting a set of prompts for individual downstream tasks. In this work, we aim to automate this prompt engineering and improve zero-shot accuracy through prompt ensembling. In particular, we ask "Given a large pool of prompts, can we automatically score the prompts and ensemble those that are most suitable for a particular downstream dataset, without needing access to labeled validation data?". We demonstrate that this is possible. In doing so, we identify several pathologies in a naive prompt scoring method where the score can be easily overconfident due to biases in pre-training and test data, and we propose a novel prompt scoring method that corrects for the biases. Using our proposed scoring method to create a weighted average prompt ensemble, our method outperforms equal average ensemble, as well as hand-crafted prompts, on ImageNet, 4 of its variants, and 11 fine-grained classification benchmarks, all while being fully automatic, optimization-free, and not requiring access to labeled validation data.

Interpreting remote sensing imagery enables numerous downstream applications ranging from land-use planning to deforestation monitoring. Robustly classifying this data is challenging due to the Earth's geographic diversity. While many distinct satellite and aerial image classification datasets exist, there is yet to be a benchmark curated that suitably covers this diversity. In this work, we introduce SATellite ImageNet (SATIN), a metadataset curated from 27 existing remotely sensed datasets, and comprehensively evaluate the zero-shot transfer classification capabilities of a broad range of vision-language (VL) models on SATIN. We find SATIN to be a challenging benchmark-the strongest method we evaluate achieves a classification accuracy of 52.0%. We provide a $\href{https://satinbenchmark.github.io}{\text{public leaderboard}}$ to guide and track the progress of VL models in this important domain.

Scene: A U.S. adversary is at work on a new type of drone, ship, or aircraft and it's your job to find it, wherever it is. Not long ago, that task would take a massive effort of human, signals, and open-source intelligence collection. But a researcher from AI company Synthetaic has created a tool that will allow users to find virtually any large object that exists in any satellite photo of the Earth within just one day. It's also the sort of capability the National Geospatial-Intelligence Agency is also looking to develop, and it could radically shift strategic advantage on the battlefield. Corey Jaskolski, founder and CEO of Synthetaic, dubbed his satellite image scanning tool Rapid Automatic Image Categorization, or RAIC.

So, using satellite photos of three cities and drawing upon methods used to manipulate video and audio files, a team of researchers set out to identify new ways of detecting fake satellite photos, warn of the dangers of falsified geospatial data and call for a system of geographic fact-checking. In 2019, the director of the National Geospatial Intelligence Agency, the organization charged with supplying maps and analyzing satellite images for the U.S. Department of Defense, implied that AI-manipulated satellite images can be a severe national security threat. When applied to the field of mapping, the algorithm essentially learns the characteristics of satellite images from an urban area, then generates a deepfake image by feeding the characteristics of the learned satellite image characteristics onto a different base map -- similar to how popular image filters can map the features of a human face onto a cat. Next, the researchers combined maps and satellite images from three cities -- Tacoma, Seattle and Beijing -- to compare features and create new images of one city, drawn from the characteristics of the other two. Low-rise buildings and greenery mark the "Seattle-ized" version of Tacoma on the bottom left, while Beijing's taller buildings, which AI matched to the building structures in the Tacoma image, cast shadows -- hence the dark appearance of the structures in the image on the bottom right.

Worries about deep fakes--machine-manipulated videos of celebrities and world leaders purportedly saying or doing things that they really didn't--are quaint compared to a new threat: doctored images of the Earth itself. China is the acknowledged leader in using an emerging technique called generative adversarial networks to trick computers into seeing objects in landscapes or in satellite images that aren't there, says Todd Myers, automation lead and Chief Information Officer in the Office of the Director of Technology at the National Geospatial-Intelligence Agency. "The Chinese are well ahead of us. This is not classified info," Myers said Thursday at the second annual Genius Machines summit, hosted by Defense One and Nextgov. "The Chinese have already designed; they're already doing it right now, using GANs--which are generative adversarial networks--to manipulate scenes and pixels to create things for nefarious reasons."

You can spend a long time searching satellite images for interesting locations. Now imagine a tool that can not only show you the location, but scans large geographical areas to find specific features that are similar, and then it presents these results in a pattern-like format. In 2002, Carnegie Mellon's (CMU) School of Computer Science launched what it calls the world's "first PhD program in Machine Learning". It attempted to learn how to program systems to automatically learn and use its experience to improve its results. A group of CMU professors and students have now created a visual search tool for satellite imagery called Terrapattern. In a landmark project, Google Earth has teamed up with locals to provide a fully immersive experience of the Sherpa community and their mountain home in the Sagarmatha region in Nepal, home to the tallest mountain in the world, Everest.

Satellites are best known for helping smartphones map driving routes or televisions deliver programs. But now, data from some of the thousands of satellites orbiting Earth are helping track things like crop conditions on rural farms, illegal deforestation, and increasingly, poverty in the hard-to-reach places around the globe. As much as that data has the potential to provide invaluable information to humanitarian organizations, watchdog groups, and policymakers, there is too much of it to sift through in order to draw insights that could influence important decisions. A team of researchers from Stanford University, however, says it has developed an efficient way. By creating a deep-learning algorithm that can recognize signs of poverty in satellite images – such as condition of roads – the team sorted through a million images to accurately identify economic conditions in five African countries, reported the scientists in the journal Science on Thursday.

Why can't computers watch the Earth from above and automatically map our roads, buildings, and trash heaps? Satellite operator DigitalGlobe is teaming up with Amazon, the venture arm of the CIA, and chipmaker Nvidia to try to make it happen. In a joint project, DigitalGlobe today released satellite imagery depicting the whole of Rio de Janeiro to a resolution of 50 centimeters. The outlines of 200,000 buildings inside the city's roughly 1,900 square kilometers have been manually marked on the photos. The SpaceNet data set, as it is called, is intended to spark efforts to train machine-learning algorithms to interpret high-resolution satellite photos by themselves.

Satellites are best known for helping smartphones map driving routes or televisions deliver programs. But now, data from some of the thousands of satellites orbiting Earth are helping track things like crop conditions on rural farms, illegal deforestation, and increasingly, poverty in the hard-to-reach places around the globe. As much as that data has the potential to provide invaluable information to humanitarian organizations, watchdog groups, and policymakers, there is too much of it to sift through in order to draw insights that could influence important decisions. A team of researchers from Stanford University, however, says it has developed an efficient way. By creating a deep-learning algorithm that can recognize signs of poverty in satellite images – such as condition of roads – the team sorted through a million images to accurately identify economic conditions in five African countries, reported the scientists in the journal Science on Thursday.